Workflow 5 - Training ImageNet CNNs on London New Build Images Label Sets

Aim

  • CNN Training Tool
  • Designed to facilitate running of Large Scale Multiple Label Set Creation and Exploration
  • Hyper Parameter and Distortion Tuning
  • CNN Training Monitoring Tools: Tensorboard Mods, Timing and Inline Confusion Matrix Rendering
  • Output: Protobuf Weights for use by CNN Labeller Script and Feature Vectors for Building Image Analysis

Method

CNN Trainer Tools

1: Quick Spin Mobile Net v2 Tensorflow Lite. Click Here

 For Testing and a quick introduction to the method and process

2: TensorBoard - CNN Training Visualisation Tool. Click Here

 Tensorflow Graphs, Traing Scalars and Distributions for best fit and archived Training Runs

3: Main CNN Training Tool. Click Here

With Tenshorhub Architectures Selector and Hypter Parameter Configuration options

4: Misclassfied Images Inspector. Click Here

 Inspect Wrongly labelled Images

5: Metrics Comparison Log Tables Click Here

 Accuracy and Speed Results on all Label Sets Trained, for all Paremter, Hypter Parameter and Architecture configurations.

6: User Modifications. Click Here

 Script Mods for easing the CNN Workflow

Convolutional Neural Networks and Transfer Learning

Key Points

  • Transfer Learning Method using Pretrained ConvNets = final layer retraining
  • Takes advantage of these learned feature maps without having to start from scratch training a large model on a large dataset.
  • Using TensorFlow Hub (image feature extraction module api) to ingest pre-trained image feature extraction module with the Inception V3 architecture trained on ImageNet (image database organized to WordNet Hierachy).
  • WordNet is a large lexical database of English. Nouns, verbs, adjectives and adverbs are grouped into sets of cognitive synonyms (synsets), each expressing a distinct concept. Synsets are interlinked by means of conceptual-semantic and lexical relations.
  • Retrain (see Tensorboard Graph below for Inception Design) the layer just before the final output layer that actually does the classification

  • Other architectures used: NASNet/PNASNet + MobileNet V1 and V2.

  • Feature Extraction uses the representations learned by a previous network to extract meaningful features from new samples. You simply add a new classifier, which will be trained from scratch, on top of the pretrained model so that you can repurpose the feature maps learned previously for our dataset.

  • In subsequent Scripts we shall use these feature maps or image feature vector(bottlenecks) to analyze images to explore similar constituent features and to search for geo-informational features specific to a London Wide Image Building Data Set.

The Inception v3 CNN Model

"Neurons that fire together, Wire Together"

  • Successor to GoogLeNet, posted a breakthrough performance on the ImageNet Visual Recognition Challenge(2014)
  • Rather than building bigger(prone to over fitting and increase in computiational resource requirement):
  • CNN 27 Layers inc Inception Layer
  • Inception Layer: Allows internal layers to choose filter size relevant to learning the required information. Allows to focus on the lexical Subject of an image, avoids learning noise.
  • https://arxiv.org/pdf/1409.4842.pdf
  • TF Dataflow graph of Inception(MobileNet version):

Transfer Learning scenarios :

http://cs231n.github.io/transfer-learning/

New dataset is small but very different from the original dataset: Since the data is small, it is likely best to only train a linear classifier. Since the dataset is very different, it might not be best to train the classifier form the top of the network, which contains more dataset-specific features. Instead, it might work better to train the SVM classifier from activations somewhere earlier in the network

New dataset is small and similar to original dataset: Since the data is small, it is not a good idea to fine-tune the ConvNet due to overfitting concerns. Since the data is similar to the original data, we expect higher-level features in the ConvNet to be relevant to this dataset as well. Hence, the best idea might be to train a linear classifier on the CNN codes.

https://www.tensorflow.org/beta/tutorials/images/transfer_learning

Do Better ImageNet Models Transfer Better? Simon Kornblith∗, Jonathon Shlens, and Quoc V. Le Google Brain {skornblith,shlens,qvl}@google.com Abstract

Activation:

The softmax layer contains N labels, this corresponds to learning N + 2048*N model parameters for the biases and weights.

Setup

In [1]:
import datetime
import os
from __future__ import print_function
from ipywidgets import interact, interactive, fixed, interact_manual
import ipywidgets as widgets

import matplotlib.pyplot as plt
import matplotlib.image as mpimg

now = datetime.datetime.now()

Skip to TensorHub Loader and Parameter Tuning CNN Trainer.

Version:

Tensorflow Version 1.8 Version 1 Signatures of Tensorhub Using TF Lite Implementation

1 - Quick Spin Trainer:

  • This Script is designed for use with the MobileNet Variations
  • Includes Quick Spin Labeller

Initialize Training Run.

Select Input and Output Folder Locations:

In [ ]:
# View Previous Set Label Run Folder
directory = '/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS'
for filename in os.listdir(directory):
      print(os.path.join(directory, filename))
In [3]:
#/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE
#ENTER TRAIN IMAGE FOLDER LOCATION
TRAIN_IMAGE_FOLDER_NAME = input("ENTER TRAIN IMAGE FOLDER LOCATION:")

ENTER TRAIN IMAGE FOLDER LOCATION:/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/TESTER
In [ ]:
directory = '/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs'
for filename in os.listdir(directory):
      print(os.path.join(directory, filename))
In [4]:
#ENTER RUN SUMMARY LOG LOCATION
RUN_FOLDER_NAME = input("ENTER RUN FOLDER NAME:")

ENTER RUN FOLDER NAME:run_215
In [38]:
# Note of Last Run Name
#run_9#run_10#run_11#run_12#run_13#run_14#run_15#run_16#run_17#run_18#run_19#run_20#run_21#run_22#run_23#run_24#run_25
#run_26,27,28
#run_91

#To Do: Automate

Enter Parameters:

In [2]:
#Load MobNet Script DEFAULTS
IMAGE_SIZE=224
ARCHITECTURE="mobilenet_0.50_" + str(IMAGE_SIZE)
SUMMARY_FOLDER = "tf_files/training_summaries"
RUN_FOLDER_NAME = now.isoformat()
STEP_SIZE = "1000"
TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/flower_photos"

# inception_v3
  • Image Size, options: '224', '192', '160', '128'. smaller = faster.

Final Parameter and Input/Output Check:

In [3]:
print(IMAGE_SIZE)
print(ARCHITECTURE)
print(TRAIN_IMAGE_FOLDER_NAME)
print(STEP_SIZE)
print(RUN_FOLDER_NAME)

224
mobilenet_0.50_224
/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/flower_photos
1000
2019-08-08T12:03:49.578712

Run MobileNet Retrain Script

Training

Once the bottlenecks are complete - examine step outputs:

  • Training Accuracy = percent of correct class
  • Validation Accuracy = performance measure not contained in the training data. Watch out for overfitting on Training Data
  • Cross Entropy = loss function/how well the learning process is progressing. The training's objective is to make the loss as small as possible, so you can tell if the learning is working by keeping an eye on whether the loss keeps trending downwards (see Tensorboard), ignoring the short-term noise.

Steps

Default =4000 Max =8000

  • Epoch: Epoch is considered as number of one pass from entire dataset
  • Steps: In tensorflow one step = number of epochs multiplied by examples divided by batch size
  • Each step chooses ten images at random from the training set, finds their bottlenecks from the cache, and feeds them into the final layer to get predictions. Those predictions are then compared against the actual labels to update the final layer's weights through the back-propagation process.
  • As the process continues we are looking for an improvement, and after all the steps are done, a final test accuracy evaluation is run on a set of images kept separate from the training and validation pictures.
  • Final Test Accuracy evaluation = Percentage of correctly labels on Test Set.
  • Varying Step Rate: The rate of improvement in the accuracy slows the longer you train for, and at some point will stop altogether (or even go down due to overfitting), but you can experiment to see what works best for your model.
In [ ]:
# Get Parameter List
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/label_image_poets.py -h
# View Script
%load /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/label_image_poets.py
In [5]:
print(IMAGE_SIZE)
print(ARCHITECTURE)
print(TRAIN_IMAGE_FOLDER_NAME)
print(STEP_SIZE)
print(RUN_FOLDER_NAME)

224
mobilenet_0.50_224
/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/flower_photos
1000
run_215
  • TensorFlow Lite defines a new model file format, based on FlatBuffers. FlatBuffers is an open-sourced, efficient cross platform serialization library. It is similar to protocol buffers, but the primary difference is that FlatBuffers does not need a parsing/unpacking step to a secondary representation before you can access data, often coupled with per-object memory allocation. Also, the code footprint of FlatBuffers is an order of magnitude smaller than protocol buffers.
In [ ]:
#Passing in Bottleneck, Training Summary and Train Image Locations as Parameters
    
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/retrain.py \
  --bottleneck_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/bottlenecks \
  --how_many_training_steps={STEP_SIZE} \
  --model_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/models/ \
  --summaries_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/training_summaries/{ARCHITECTURE} \
  --output_graph=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_graph.pb \
  --output_labels=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_labels.txt \
  --architecture={ARCHITECTURE} \
  --image_dir={TRAIN_IMAGE_FOLDER_NAME}

print('jupyter --> done')
print(now.isoformat())

Go to to Tensorboard

Dont Forget to Log Training Run Details



Test/Train Split and Cross Validation in Tensorflow:

One of the things the (Tensorflow Hub) script does under the hood when you point it at a folder of images is divide them up into three different sets. The usual split is to put 80% of the images into the main training set, keep 10% aside to run as validation frequently during training, and then have a final 10% that are used less often as a testing set to predict the real-world performance of the classifier. These ratios can be controlled using the --testing_percentage and --validation_percentage flags. In general you should be able to leave these values at their defaults, since you won't usually find any advantage to training to adjusting them.

Note that the script uses the image filenames (rather than a completely random function) to divide the images among the training, validation, and test sets. This is done to ensure that images don't get moved between training and testing sets on different runs, since that could be a problem if images that had been used for training a model were subsequently used in a validation set.

You might notice that the validation accuracy fluctuates among iterations. Much of this fluctuation arises from the fact that a random subset of the validation set is chosen for each validation accuracy measurement. The fluctuations can be greatly reduced, at the cost of some increase in training time, by choosing --validation_batch_size=-1, which uses the entire validation set for each accuracy computation.

Once training is complete, you may find it insightful to examine misclassified images in the test set. This can be done by adding the flag --print_misclassified_test_images. This may help you get a feeling for which types of images were most confusing for the model, and which categories were most difficult to distinguish. For instance, you might discover that some subtype of a particular category, or some unusual photo angle, is particularly difficult to identify, which may encourage you to add more training images of that subtype. Oftentimes, examining misclassified images can also point to errors in the input data set, such as mislabeled, low-quality, or ambiguous images. However, one should generally avoid point-fixing individual errors in the test set, since they are likely to merely reflect more general problems in the (much larger) training set.

https://www.tensorflow.org/hub/tutorials/image_retraining

Quick Spin Label Tester

View Label Folders

In [20]:
# View Folders from last Label Run
directory = TRAIN_IMAGE_FOLDER_NAME
for filename in os.listdir(directory):
      print(os.path.join(directory, filename))

/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE/.DS_Store
/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE/FLAT
/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE/HOUSE
/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE/INDUSTRIAL
/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE/OFFICE

Select Label Folder:

In [21]:
#ENTER TEST IMAGE FOLDER
TEST_FOLDER_NAME = input("ENTER TEST IMAGE FOLDER LOCATION: ")

ENTER TEST IMAGE FOLDER LOCATION: /Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE/OFFICE
In [ ]:
directory =  TEST_FOLDER_NAME
for filename in os.listdir(directory):
      print(os.path.join(directory, filename))

Select Image:

In [28]:
#ENTER TRAIN SUMMARY LOG LOCATION
TEST_IMAGE = input("ENTER TEST LABEL IMAGE NAME:")

ENTER TEST LABEL IMAGE NAME:/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE/OFFICE/col_id_57146.jpg
In [29]:
img=mpimg.imread(TEST_IMAGE)

# Get current size
fig_size = plt.rcParams["figure.figsize"]
 
# Prints: [8.0, 6.0]
print ("Current size:", fig_size)
 
# Set figure width to 12 and height to 9
fig_size[0] = 12
fig_size[1] = 9
plt.rcParams["figure.figsize"] = fig_size

plt.imshow(img)

Current size: [12.0, 9.0]
Out[29]:
<matplotlib.image.AxesImage at 0x11ee00a20>

Note:

The script will write out the new model trained on your categories to /tmp/output_graph.pb, and a text file containing the labels to /tmp/output_labels.txt. The new model contains both the TF-Hub module inlined into it, and the new classificiation layer. The two files are both in a format that the C++ and Python image classification examples can read in, so you can start using your new model immediately. Since you've replaced the top layer, you will need to specify the new name in the script, for example with the flag --output_layer=final_result if you're using label_image.

Here's an example of how to run the label_image example with your retrained graphs. By convention, all TensorFlow Hub modules accept image inputs with color values in the fixed range [0,1], so you do not need to set the --input_mean or --input_std flags.

Execute Classify Image Script on Sample Image:

In [31]:
print({RUN_FOLDER_NAME})
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/label_image_poets.py \
--graph=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_graph.pb  \
--labels=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_labels.txt  \
--image={TEST_IMAGE}

{'run_92'}

Evaluation time (1-image): 0.204s

office (score=0.98553)
industrial (score=0.01290)
house (score=0.00148)
flat (score=0.00009)
In [ ]:
# Get Parameter List
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/label_image_poets.py -h
# View Script
%load /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/label_image_poets.py

Brief description of the Script:

  • Function Map
  • Transfer Learning

  • Last Layer Regression

  • Parameter Setup

  • Download Mdule from Hub
  • Image Decoding and Processing
  • Cache Bottlenecks
  • Last Layer Training
  • Tensorflow Tensors and Graphs
  • Summary Ops Logging for Tenosrboard and Realtime Metrics
  • Managing Inputs and Output of Weights Model

2 - Tensorboard

TensorBoard is a suite of web applications for inspecting and understanding your TensorFlow runs and graphs.

Summary Ops:

How TensorBoard gets data from TensorFlow

The first step in using TensorBoard is acquiring data from your TensorFlow run. For this, you need summary ops. Summary ops are ops, like tf.matmul or tf.nn.relu, which means they take in tensors, produce tensors, and are evaluated from within a TensorFlow graph. However, summary ops have a twist: the Tensors they produce contain serialized protobufs, which are written to disk and sent to TensorBoard. To visualize the summary data in TensorBoard, you should evaluate the summary op, retrieve the result, and then write that result to disk using a summary.FileWriter. A full explanation, with examples, is in the tutorial.

Runs:

Comparing different executions of your model

You may want to visually compare multiple executions of your model; for example, suppose you've changed the hyperparameters and want to see if it's converging faster. TensorBoard enables this through different "runs". When TensorBoard is passed a logdir at startup, it recursively walks the directory tree rooted at logdir looking for subdirectories that contain tfevents data. Every time it encounters such a subdirectory, it loads it as a new run, and the frontend will organize the data accordingly.

Bottleneck Location Selector:

  • Use this Widget to load up and Analyze data from the previous Training Runs Archive
In [ ]:
import ipywidgets as widgets

w = widgets.IntSlider(max=300)
display(w)

#To Do: Use OS Lib to get Run Count/Run Folder names
# Click Mouse + Click Arrow Keys for Accuracy

In [41]:
RUN_FOLDER_NAME = "run_" + str(w.value)
print(RUN_FOLDER_NAME)

run_22

Go to to MobNet Only Trainer Cells

Go to to CNN Hub Selector Trainer and Modified Trainer Cells

Link to TensorHub Version Trainer Cells

In [42]:
from IPython.core.display import display, HTML

#VM Commands:
#cd $HOME/ml2
#source env/bin/activate

#Copy the Following into Command:
print("tensorboard --logdir /Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/"  
      + RUN_FOLDER_NAME + "/training_summaries")
display(HTML("<a href='http://localhost:6006' target='_blank'>LAUNCH TENSORBOARD (in new tab)</a>"))

tensorboard --logdir /Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/run_22/training_summaries
LAUNCH TENSORBOARD (in new tab)

Flowers BenchMark:

London Development Database, Best Fit Label Set:

Tensorhub Model Architecture Comparison, Best Fit Label Set:

Tensorboard Hub:

Full List of Model Architectures available on the hub: https://tfhub.dev/s?module-type=image-feature-vector

Signature: The signature of the module specifies what is the purpose for which module is being used for. All the module, comes with the ‘default’ signature and makes use of it, if a signature is not explicitly mentioned. For most of the modules, when ‘default’ signature is used, the internal layers of the model is abstracted from the user. The function used to list out all the signature names of the module is get_signature_names().

Each of the module has some set of expected inputs depending upon the signature of the module being used. Though most of the modules have documented the set of expected inputs in TensorFlow Hub website (particularly for ‘default’ signature), some of them haven’t. In this case, it is easier to obtain the expected input along with it’s size and datatype using the get_input_info_dict().

Expected outputs: In order to build the remaining part of the graph after the TensorFlow Hub’s model is built, it is necessary to know the expected type of output. get_output_info_dict() function is used for this purpose. Note that for ‘default’ signature, there will be usually only one output, but when you use a non-default signature, multiple layers of the graph will be exposed to you.

3 - Main CNN Training Tool (with Tensor Hub Selector and Hyper Parameter Variation)

  • Version 1 Modules (Tensorflow 1.12 from 2018)
  • Uses TFLite
  • Later Versions wil not work as they are not built on TFLite Framework
  • Check the Usage Documentation on TF Hub esp Signature(eg Vector vs Classification and Inputs and Outputs). For instance, num_features and height x width params might need changin, depending on how the orginal model was built(eg where size of image is important to architectures design aim e.g. speed vs accuracy). Original scripts dont accomdate this change, so you have to edit the script ourselves.
  • Usage Docs also outline areas for for Fine Tuning: https://tfhub.dev/google/imagenet/nasnet_mobile/feature_vector/3
In [9]:
#ENTER RUN SUMMARY LOG LOCATION
RUN_FOLDER_NAME = "run_" + input("Run Folder:")

Run Folder:119

</div>

In [ ]:
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/retrain_full.py \
--tfhub_module={MODULE} \
--image_dir={TRAIN_IMAGE_FOLDER_NAME} \
--saved_model_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/models \
--bottleneck_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/bottlenecks \
--how_many_training_steps={STEP_SIZE} \
--summaries_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/training_summaries/{ARCHITECTURE} \
--output_graph=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_graph.pb \
--output_labels=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_labels.txt
In [ ]:
#Label Run
w2 = widgets.IntSlider(max=300, continuous_update=True)
display(w2)
In [4]:
# Check Parameter Options
# %run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/retrain_full.py -h

usage: retrain_full.py [-h] [--image_dir IMAGE_DIR] [--output_graph OUTPUT_GRAPH] [--intermediate_output_graphs_dir INTERMEDIATE_OUTPUT_GRAPHS_DIR] [--intermediate_store_frequency INTERMEDIATE_STORE_FREQUENCY] [--output_labels OUTPUT_LABELS] [--summaries_dir SUMMARIES_DIR] [--how_many_training_steps HOW_MANY_TRAINING_STEPS] [--learning_rate LEARNING_RATE] [--testing_percentage TESTING_PERCENTAGE] [--validation_percentage VALIDATION_PERCENTAGE] [--eval_step_interval EVAL_STEP_INTERVAL] [--train_batch_size TRAIN_BATCH_SIZE] [--test_batch_size TEST_BATCH_SIZE] [--validation_batch_size VALIDATION_BATCH_SIZE] [--print_misclassified_test_images] [--bottleneck_dir BOTTLENECK_DIR] [--final_tensor_name FINAL_TENSOR_NAME] [--flip_left_right] [--random_crop RANDOM_CROP] [--random_scale RANDOM_SCALE] [--random_brightness RANDOM_BRIGHTNESS] [--tfhub_module TFHUB_MODULE] [--saved_model_dir SAVED_MODEL_DIR]

# Launch Tensorboard # tensorboard --logdir /Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/run_100/training_summaries

Link to TensorBoard Cells

Load Defaults

In [4]:
#Load MobNet Script DEFAULTS
IMAGE_SIZE=224
ARCHITECTURE="mobilenet_0.50_" + str(IMAGE_SIZE)
SUMMARY_FOLDER = "tf_files/training_summaries"
RUN_FOLDER_NAME = now.isoformat()
STEP_SIZE = "1000"
TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/flower_photos"
In [19]:
# Manual Insertion of Parameters:
# To Do: ReCode Cell as User Component

ARCHITECTURE = "mobnetv1"
MODULE = "https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/feature_vector/1"
#ARCHITECTURE = "mobnetv2"
#MODULE = "https://tfhub.dev/google/imagenet/mobilenet_v2_100_224/feature_vector/1"
#mobnet 140 vs 35
#ARCHITECTURE = "mobnetv2_35"    
#MODULE = "https://tfhub.dev/google/imagenet/mobilenet_v2_035_224/feature_vector/1"
#ARCHITECTURE = "mobnetv2_140"
#MODULE = "https://tfhub.dev/google/imagenet/mobilenet_v2_140_224/feature_vector/1"
#ARCHITECTURE = "mobnet_class_1"
#MODULE = "https://tfhub.dev/google/imagenet/mobilenet_v2_140_224/classification/1"
#ARCHITECTURE = "mobnet_class"
#MODULE = "https://tfhub.dev/google/imagenet/mobilenet_v2_140_224/classification/1"
#ARCHITECTURE = "mobnet_quant"
#MODULE = "https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/quantops/feature_vector/1" 
#ARCHITECTURE = "inception_v3"
#MODULE = "https://tfhub.dev/google/imagenet/inception_v3/feature_vector/1"
#ARCHITECTURE = "nasnet_mob"
#MODULE = "https://tfhub.dev/google/imagenet/nasnet_mobile/feature_vector/1"
#ARCHITECTURE = "nasnet_large"
#MODULE = "https://tfhub.dev/google/imagenet/nasnet_large/feature_vector/1"
#ARCHITECTURE = "resNet_v1_small"
#MODULE = "https://tfhub.dev/google/imagenet/resnet_v1_50/feature_vector/1"
#ARCHITECTURE = "resNet_v1"
#MODULE = "https://tfhub.dev/google/imagenet/resnet_v1_152/feature_vector/1"
#ARCHITECTURE = "resNet_v2"
#MODULE = "https://tfhub.dev/google/imagenet/resnet_v2_152/feature_vector/1"
#ARCHITECTURE = "inception_resnet"
#MODULE = "https://tfhub.dev/google/imagenet/inception_resnet_v2/feature_vector/1"
#ARCHITECTURE = "pnasnet"
#MODULE = "https://tfhub.dev/google/imagenet/pnasnet_large/feature_vector/1"


TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_56/train_1K"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/flower_photos"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_6_2"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/ldd_newbuild_images_arch"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_71/TEST"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_19"

#One vs Rest
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_71/TEST"

#Munged Hybrid
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_72/TRAIN"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_72/TERRACES_HOUSES/TRAIN"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_85/TRAIN"

#--random_crop=5 \
#--random_scale=5 \
#--print_misclassified_test_images
#--random_brightness=5 \
#--flip_left_right  No!
#--learning_rate
#--train_batch_size
#--random_brightness=5 \
#--saved_model_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/models \
#--bottleneck_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/bot_cache/bottlenecks \

STEP_SIZE = "1000"
RUN_FOLDER_NAME = "run_227"
para_run = 1
In [20]:
#RUN_FOLDER_NAME = str(w2.value)
### Check Parameters
print('para_run  ' + str(para_run))
print('IMAGE_SIZE  ' + str(IMAGE_SIZE))
print('STEP_SIZE  ' + str(STEP_SIZE))
print('ARCHITECTURE  ' + str(ARCHITECTURE))
print('MODULE  ' + str(MODULE))
print('TRAIN_IMAGE_FOLDER_NAME  ' + str(TRAIN_IMAGE_FOLDER_NAME))
print('RUN_FOLDER_NAME  ' + str(RUN_FOLDER_NAME))

para_run  1
IMAGE_SIZE  224
STEP_SIZE  1000
ARCHITECTURE  mobnetv1
MODULE  https://tfhub.dev/google/imagenet/mobilenet_v1_100_224/feature_vector/1
TRAIN_IMAGE_FOLDER_NAME  /Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE
RUN_FOLDER_NAME  run_227
In [ ]:
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/retrain_full.py \
--tfhub_module={MODULE} \
--how_many_training_steps={STEP_SIZE} \
--saved_model_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/models_{now.isoformat()} \
--image_dir={TRAIN_IMAGE_FOLDER_NAME} \
--print_misclassified_test_images \
--summaries_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/training_summaries/{ARCHITECTURE}/{para_run} \
--output_graph=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_graph.pb \
--output_labels=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_labels.txt 


#print(now.isoformat())
#test_var
#--saved_model_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/models_{now.isoformat()} \

4 - Inspect Misclassified Thumb Gallery

In [2]:
#Small Mod to Allow us to Import Misclassifieds into an Image Gallery
import csv

with open('misclass.csv', 'r') as f:
  reader = csv.reader(f)
  mis_class = list(reader)

In [2]:
## Load Librarys and Jupyter User Settings
from IPython.core.display import display, HTML
display(HTML("<style>.container { width:760 !important; }</style>"))
from IPython.display import Markdown, display, HTML
from IPython.display import Image as Img

def css_styling():
    styles = open('custom_thumbs.css', 'r').read()
    return HTML(styles)

css_styling()
Out[2]:
In [ ]:
gallery = ""
for a in mis_class:
    #print(str(a[0])[115:])
    #print(str(a[1]))
    inp3 = "<div class='gallery'><div class='zoom'><img src='../../" + str(a[0][25:]) + "' width='' height=''></div><div class='desc'> " + str(a[0][124:]) + " Misclassified as " + str(a[1]) + ". G_Truth is " + str(a[2]) + "<br>Img = " + str(a[0][78:98]) +"</div></div>"
    gallery = gallery + inp3

display(HTML(gallery))

Sample Thumb Gallery Runs for Technical Write Up

In [5]:
#Small Mod to Allow us to Import Misclassifieds into an Image Gallery
import csv

with open('trueclass.csv', 'r') as f:
  reader = csv.reader(f)
  mis_class = list(reader)
In [ ]:
#Version 3
gallery = ""

for a in mis_class:
    #print(str(a[0])[80:])
    #print(str(a[1]))
    inp3 = "<div class='gallery'><div class='zoom'><img src='../../" + str(a[0][25:]) + "' width='' height=''></div><div class='desc'> " + str(a[0][124:]) + "Correct G_Truth is " + str(a[2]) + "<br>Img = " + str(a[0][78:100]) +"</div></div>"
    gallery = gallery + inp3

display(HTML(gallery))
In [ ]:
#Version 3
gallery = ""

for a in mis_class:
    #print(str(a[0])[79:])
    #print(str(a[1]))
    inp3 = "<div class='gallery'><div class='zoom'><img src='../../" + str(a[0][25:]) + "' width='' height=''></div><div class='desc'> " + str(a[0][79:]) + " Misclassified as " + str(a[1]) + ". G_Truth is " + str(a[2]) + "</div></div>"
    gallery = gallery + inp3

display(HTML(gallery))
In [ ]:
#Version 2
gallery = ""

for a in mis_class:
    #print(str(a[0])[79:])
    #print(str(a[1]))
    inp3 = "<div class='gallery'><div class='zoom'><img src='../../" + str(a[0][25:]) + "' width='' height=''></div><div class='desc'> " + str(a[0][79:]) + " Misclassified as " + str(a[1]) + ". G_Truth is " + str(a[2]) + "</div></div>"
    gallery = gallery + inp3

display(HTML(gallery))
In [ ]:
#Version 1
gallery = ""

for a in mis_class:
    #print(str(a[0])[79:])
    #print(str(a[1]))
    inp3 = "<div class='gallery'><div class='zoom'><img src='../../" + str(a[0][25:]) + "' width='' height=''></div><div class='desc'> " + str(a[0][79:]) + " Misclassified as " + str(a[1]) + ". G_Truth is " + str(a[2]) + "</div></div>"
    gallery = gallery + inp3

display(HTML(gallery))

NOTE:

For a more in depth exploration of Ground Truth and Prediction Discrepancies, and thematic treatment of the urban design issues raised, please proceed to the CNN Classification, Map and Image Analysis Workflow Scripts.

Inspect Mis Classified Images

In [37]:
TEST_IMAGE = '/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_68/FINAL_LABELS/TRAIN/HOUSE_FLATS_COMMERCIAL/HOUSE/col_id_20092.jpg'
In [38]:
img=mpimg.imread(TEST_IMAGE)

# Get current size
fig_size = plt.rcParams["figure.figsize"]
 
# Prints: [8.0, 6.0]
print ("Current size:", fig_size)
 
# Set figure width to 12 and height to 9
fig_size[0] = 12
fig_size[1] = 9
plt.rcParams["figure.figsize"] = fig_size

plt.imshow(img)

Current size: [12.0, 9.0]
Out[38]:
<matplotlib.image.AxesImage at 0x12536e278>

Quick label Test

In [10]:
#ENTER TRAIN SUMMARY LOG LOCATION
#TEST_IMAGE = input("ENTER TEST LABEL IMAGE NAME:")
TEST_IMAGE = '/Users/anthonysutton/ml2/_STREETVIEW_EXPLORER/LDD_Complete/col_id_17930.jpg'
In [11]:
#Check you are using Correct Model Input Parameters 
#ht = 224
#wd = 224
#--input_layer=Placeholder \
#--output_layer=final_result \
#--input_height={ht} --input_width={wd} \
In [12]:
print({RUN_FOLDER_NAME})
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/label_image_full.py \
--graph=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_graph.pb  \
--labels=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_labels.txt  \
--input_layer=Placeholder \
--output_layer=final_result \
--image={TEST_IMAGE}

{'run_122'}
flat 0.730731
office 0.25494254
house 0.012470891
industrial 0.001855493
In [21]:
TEST_IMAGE
Out[21]:
'/Users/anthonysutton/ml2/_FINAL_SCRIPT_LIBRARY/LABELS/LABEL_RUN_2_2_SAFE/FLAT/col_id_14360.jpg'
In [22]:
img=mpimg.imread(TEST_IMAGE)

# Get current size
fig_size = plt.rcParams["figure.figsize"]
 
# Prints: [8.0, 6.0]
print ("Current size:", fig_size)
 
# Set figure width to 12 and height to 9
fig_size[0] = 12
fig_size[1] = 9
plt.rcParams["figure.figsize"] = fig_size

plt.imshow(img)

Current size: [12.0, 9.0]
Out[22]:
<matplotlib.image.AxesImage at 0x11e1a3550>

5 - CNN Trainer Metrics Log Tables

Best Fit Label Set Variation Scores:

Model Architectures and Hyper Parameter Scores:

Summary

  • Bottleneck Creations (extraction of penultimate later of image features) is relatively time-expensive
  • Google Dataflow API addreses this workflow issue by processing each image independently and in parallel
  • We assess this benefits of this approach in the Google Cloud ML Implementation of our workflow, in the final scripts.

Notes

  • While training our model we found refraining from exporting the model function (model is only needed once we strat using the classification). Bottlenecks were also pointed to same folder, so can be reused when not neccessary to regernearte(eg repeat runs of same archtiecture).

Saving Models

  • The entire model can be saved to a file that contains the weight values, the model's configuration, and even the optimizer's configuration (depends on set up). This allows you to checkpoint a model and resume training later.

  • Fully-functional saved models can be loaded into TensorFlow.js (HDF5, Saved Model) and then train and run them in web browsers, or convert them to run on mobile devices using TensorFlow Lite (HDF5, Saved Model)

  • Using the Scalar Tool during training to compare diff model builds. Toggle Run button. Visual comparison of Training process.

Architecture Exploration Notes:

  • Accuracy vs Latency
  • Speed to Converge

Appendix 1 - User Modified Trainer Scripts

Timer and Confusion Matrix Modifications

  • we made modifications to the Tensorflow Scripts to aid with collating the Model benchmark Performance Data for our final Accuracy Matrix, and for moving through the project workflow in general.

Link to TensorBoard Cells

Load Default Parameters

In [ ]:
# These are all parameters that are tied to the particular model architecture
# we're using for Inception v3. These include things like tensor names and their
# sizes. If you want to adapt this script to work with another model, you will
# need to update these to reflect the values in the network you're using.
In [46]:
#LOAD DEFAULTS
IMAGE_SIZE=224
ARCHITECTURE="mobilenet_0.50_" + str(IMAGE_SIZE)
SUMMARY_FOLDER = "tf_files/training_summaries"
#RUN_FOLDER_NAME = now.isoformat()
RUN_FOLDER_NAME = "run_test"
STEP_SIZE = "50"
#STEP_SIZE = "1000"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/flower_photos"
#TRAIN_IMAGE_FOLDER_NAME = "/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/ldd_newbuild_images_arch"
In [47]:
RUN_FOLDER_NAME = 'run_166'
In [48]:
### Check Parameters
print(IMAGE_SIZE)
print(ARCHITECTURE)
print(TRAIN_IMAGE_FOLDER_NAME)
print(STEP_SIZE)
print(RUN_FOLDER_NAME)

224
mobilenet_0.50_224
/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/ldd_newbuild_images_arch
50
run_166

5.1 - Confusion Matrix Only Version

In [ ]:
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/retrain_confusion_plot.py \
  --bottleneck_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/bottlenecks \
  --how_many_training_steps={STEP_SIZE} \
  --model_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/models/ \
  --summaries_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/training_summaries/{ARCHITECTURE} \
  --output_graph=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_graph.pb \
  --output_labels=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_labels.txt \
  --architecture={ARCHITECTURE} \
  --image_dir={TRAIN_IMAGE_FOLDER_NAME}

print('jupyter --> done')
print(now.isoformat())

  • Tensorboard Mod ScreenShot:
In [ ]:
#Input image resolution: 128,160,192, or 224px. Unsurprisingly, feeding in a higher resolution image takes more processing time, but results in better classification accuracy.
#The relative size of the model as a fraction of the largest MobileNet: 1.0, 0.75, 0.50, or 0.25.
    
#IMAGE_SIZE=128,160,192, or 224px
#ARCHITECTURE="mobilenet_0.50_${IMAGE_SIZE}"

#inception_v3

# Link to Different Models
#https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/detection_model_zoo.md

5.2 - Timer Only Version

In [ ]:
%run -i /Users/anthonysutton/ml2/tensorflow-for-poets-2/scripts/retrain_timer.py \
  --bottleneck_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/bottlenecks \
  --how_many_training_steps={STEP_SIZE} \
  --model_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/models/ \
  --summaries_dir=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/training_summaries/{ARCHITECTURE} \
  --output_graph=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_graph.pb \
  --output_labels=/Users/anthonysutton/ml2/tensorflow-for-poets-2/tf_files/train_runs/{RUN_FOLDER_NAME}/retrained_labels.txt \
  --architecture={ARCHITECTURE} \
  --image_dir={TRAIN_IMAGE_FOLDER_NAME}

print('jupyter --> done')
print(now.isoformat())